Memory-aware and context-aware multi-DNN inference on the edge

Deep neural networks (DNNs) are becoming the core components of many applications running on edge devices, especially for real time image-based analysis. Increasingly, multi-faced knowledge is extracted by executing multiple DNNs inference models, e.g., identifying objects, faces, and genders from images. It paramount importance to guarantee low response times such multi-DNN executions as it affects not only users quality experience but also safety. The challenge, largely unaddressed state art, how overcome memory limitation devices without altering DNN models. In this paper, we design implement Masa , a responsive memory-aware execution scheduling framework, which requires no modification aim consistently ensure average when deterministically stochastically DNN-based image analyses. enabling features (i) modeling inter- intra-network dependency, (ii) leveraging complimentary usage each layer, (iii) exploring context dependency DNNs. We verify correctness optimality via mixed integer programming. extensively evaluate two versions context-oblivious context-aware, three configurations Raspberry Pi large set popular models triggered different generation patterns Our evaluation results show that can achieve lower up 90% with small memory, i.e., 512 MB 1 GB, compared art solutions.